Educational testing and validity of conclusions in the scholarship of teaching and learning.

Validity and its integral evidence of reliability are fundamentals for educational and psychological measurement, and standards of educational testing. Herein, we describe these standards of educational testing, along with their subtypes including internal consistency, inter-rater reliability, and inter-rater agreement. Next, related issues of measurement error and effect size are discussed. This article concludes with a call for future authors to improve reporting of psychometrics and practical significance with educational testing in the pharmacy education literature. By increasing the scientific rigor of educational research and reporting, the overall quality and meaningfulness of SoTL will be improved.

An experimental study using Rasch analysis to compare absolute magnitude estimation and categorical rating scaling as applied in survey research.

Limited research has applied a measurement model to compare the rating scale functioning of categorical rating scaling (CRS) and absolute magnitude estimation scaling (MES) when rating subjective stimuli. We used an experimental design and applied the Rasch model to the survey data, with each respondent rating items using MES and one of four commonly used agreement-disagreement rating scales. The results indicated that the CRS and MES data were comparable in person and item separation and reliability when the respondents' scales were known. MES had lower standard error for people and items; however MES had disordered step calibrations. Finally, the respondents reported preference of CRS to MES.

Effects of compliance monitoring of vocal function exercises on voice outcome measures for normal voice.

This experimental study examined the efficacy of the vocal function exercise program in improving voice production in individuals with normal voices. 20 young women (M age = 22 yr.) with normal voices, 10 in the Monitored compliance group and 10 in the Unmonitored compliance group, received training in performing vocal function exercises followed by daily practice of the exercises for 28 consecutive days. Participants in the Monitored compliance group were required to submit audio or video recordings of their daily practice of vocal function exercises, and those in the Unmonitored compliance group were not required to record their daily practice sessions. Results indicated that while the participants in both groups significantly increased maximum phonation times and maximum phonational frequency ranges, those in the Monitored group improved significantly more than the participants in the Unmonitored group on these outcome measures.

A computer-based module for prescribing error instruction.

OBJECTIVE: To evaluate an instructional module's effectiveness at changing third-year doctor of pharmacy (PharmD) students' ability to identify and correct prescribing errors. DESIGN: Students were randomized into 2 groups. Using a computer-based module, group 1 completed worksheet A, watched a presentation on medication errors, and then completed worksheets B and C. Group 2 completed worksheets A and B, watched the presentation, and then completed worksheet C. ASSESSMENT: Both groups scored a median 50% on worksheet A and 66.7% on worksheet C (p < 0.001). Median scores on worksheet B differed between groups (p = 0.0014). Group 1 viewed the presentation before completing worksheet B and scored 62.5%, while group 2 viewed the presentation after scoring 50% on worksheet B. CONCLUSION: The module effectively taught pharmacy students to identify and correct prescribing errors.

Rasch analysis of word identification and magnitude estimation scaling responses in measuring naive listeners' judgments of speech intelligibility of children with severe-to-profound hearing impairments.

PURPOSE: Speech intelligibility research typically relies on traditional evidence of reliability and validity. This investigation used Rasch analysis to enhance understanding of the functioning and meaning of scores obtained with 2 commonly used procedures: word identification (WI) and magnitude estimation scaling (MES). METHOD: Narrative samples of children with hearing impairments were used to collect data from listeners with no previous experience listening to or judging intelligibility of speech. WI data were analyzed with the Rasch rating scale model. MES data were examined with Rasch partial credit model when individual scales were unknown, and the Rasch rating scale model was used with reported individual scales. RESULTS: Results indicated that both procedures have high reliability and similar discriminatory power. However, reliability and separation were lower for MES when scales were unknown. Both procedures yielded similar speech sample ordering by their difficulty. However, sampling gaps were noted as well as item misfit issues. CONCLUSIONS: Functioning wise, both WI and MES procedures were highly reliable in measuring speech intelligibility, and measurement precision may be increased by asking participants to report their individual scales when using MES. Meaning wise, operationalization of speech intelligibility did not change when either WI or MES procedure was used. However, the sample selection procedure needs to be further refined to allow for a wider selection of stimuli.

Effects of training on naïve listeners' judgments of the speech intelligibility of children with severe-to-profound hearing loss.

PURPOSE: This study examined the effects of feedback training, familiarization training, and no training on naïve listeners' word identification (WI) and magnitude estimation scaling (MES) judgments of the speech intelligibility of children with severe-to-profound hearing impairments. METHOD: Depending on the training group, listeners received a pretest, an immediate posttest, and/or a delayed posttest. RESULTS: Results indicated that repeated exposure, with or without training, led to improved WI scores. Beyond the effects of repeated exposure, listeners' WI judgments of the intelligibility of speech significantly increased immediately after training in which listeners received feedback regarding the accuracy of their WI responses. The MES results were less straightforward-listeners in the feedback group perceived speech samples as less intelligible after the training, perceptions of speech intelligibility stayed almost the same for the familiarization training group, and participants in the control group perceived speech samples as more intelligible at the posttest. For the training groups that were not pretested, perceptions improved from the immediate to delayed posttest. DISCUSSION: Results may have both theoretical and clinical significance, particularly as they relate to contrasting theories of perceptual learning and the extent to which listener characteristics may be reflected in intelligibility judgments.

Magnitude estimation and categorical rating scaling in social sciences: a theoretical and psychometric controversy.

This paper revisits a half-century long theoretical controversy associated with the use of magnitude estimation scaling (MES) and category rating scaling (CRS) procedures in measurement. The MES procedure in this study involved instructing participants to write a number that matched their impression of difficulty of a test item. Participants were not restricted in the range of numbers they could choose for their scale. They also had the choice of disclosing their individual scale. After the MES task was completed, participants were given a blank copy of the test to rate the perceived difficulty of each item using a researcher-imposed categorical rating scale from 1 (very easy) to 6 (very difficult). The MES and CRS data were both analyzed using Rasch Rating scale model. Additionally, the MES data were examined with Rasch Partial Credit model. Results indicate that knowing each person's scale is associated with smaller errors of measurement.

Deconstructing therapy outcome measurement with Rasch analysis of a measure of general clinical distress: the Symptom Checklist-90-Revised.

Rasch analysis was used to illustrate the usefulness of item-level analyses for evaluating a common therapy outcome measure of general clinical distress, the Symptom Checklist-90-Revised (SCL-90-R; Derogatis, 1994). Using complementary therapy research samples, the instrument's 5-point rating scale was found to exceed clients' ability to make reliable discriminations and could be improved by collapsing it into a 3-point version (combining scale points 1 with 2 and 3 with 4). This revision, in addition to removing 3 misfitting items, increased person separation from 4.90 to 5.07 and item separation from 7.76 to 8.52 (resulting in alphas of .96 and .99, respectively). Some SCL-90-R subscales had low internal consistency reliabilities; SCL-90-R items can be used to define one factor of general clinical distress that is generally stable across both samples, with two small residual factors.

Equating student satisfaction measures.

Colleges and universities conduct student satisfaction studies for many important policy making reasons. However the differences in instrumentation and the use of students' self-reported ratings of satisfaction makes such decisions sample-, instrument-, and institution-dependent. A common metric of student satisfaction would assist decision makers by providing a richness of information not typically obtained. The present study investigated the extent to which two nationally known instruments of student satisfaction could be scaled on the same quantitative metric. Pseudo-common item equating (Fisher, 1997) based on five link items of low and high endorsability enabled comparisons of "similar, but not identical items, from different instruments, calibrated on different samples" (p. 87). Results suggest that both instruments measured similar constructs and could be reasonably used to create a single, common metric. While samples used in the experiment were less than ideal, results clearly demonstrated the usefulness and reasonability of the pseudo-common item equating process.